Disk subsystems and their integrated system

ABSTRACT

An asynchronous remote copy system which can ensure the data renewal order and data integrity of the disk subsystems. The system provides for data mirroring by a main center having a gateway and a remote center having a gateway, and storage systems in each center connected to the corresponding gateway. Data is mirrored through synchronous type remote copy between volumes of the storage systems of each center. The gateway of the main center sends the renewal data to the gateway of the remote center in accordance with the order of renewal of volumes, to make the gateway of the remote center reflect the renewal data upon the volumes thereof through asynchronous type remote copy.

The present application is a continuation of application Ser. No.09/513,932, filed Feb. 28, 2000, now U.S. Pat. No. 7,167,902, thecontents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

The present invention relates to external storage devices for storingdata in a computer system and to their integrated system, and moreparticularly to remote copy techniques for mirroring data between remoteexternal storage devices (disk subsystems) without involving an upperhierarchical apparatus or host computer, by interconnecting remoteexternal storage devices and other remote external storage devices. Thedisk subsystem is herein intended to mean a control unit for controllingdata transfer to and from an upper hierarchical apparatus and a storagedevice having disks for storing data or a storage device having aninternal buffer.

External storage systems incorporating a so-called remote copy functionhave already been in practical use, in which data is mirrored and storedin disk subsystems of a main center and a remote center.

Such prior art has various issues to be solved because the remote copyfunction is realized by involving host computers.

“Synchronous Type and Asynchronous Type”

The remote copy function is mainly classified into two types, asynchronous type and an asynchronous type.

The synchronous type executes the following process sequence. When adisk subsystem is instructed by a host computer (upper hierarchicalapparatus) of a main center to renew (write) data and if the disksubsystem is assigned the remote copy function, a renewal processcompletion notice is issued to the host computer of the main center onlyafter the instructed data renewal (write) is completed for acorresponding disk subsystem in a remote center. A time delay(transmission time and the like) is generated in accordance with ageographical distance between the main center and remote center and theperformance of a data transmission line therebetween. If thetransmission time of the synchronous type is taken into consideration,several tens Km is a practical limit of a distance to a remote site.

In the synchronous type, the data contents in disk subsystems in themain and remote centers are always consistent from a macro viewpoint.Therefore, even if the function of the main center is lost by accidentsor the like, the data contents immediately before the accidents areperfectly retained in the disk subsystems of the remote center and theprocess can be resumed quickly at the remote center. The term “alwaysconsistent from the macro viewpoint” means that during the execution ofthe synchronous type function, although the data contents may bedifferent in terms of a process time (•sec, msec) of magnetic diskdevices and electronic circuits, the data contents are always the sameat the time of data renewal completion. This is because the renewalprocess at the main center cannot be completed unless the renewal datais completely reflected upon the remote center. Therefore, in somecases, particularly if a distance between the main and remote centers islong and the data transmission line is congested, the access performanceto a disk subsystem in the main center is considerably degraded.

In contrast, the asynchronous type executes the following processsequence. When a disk subsystem is instructed by a host computer of amain center to renew (write) data and even if this data is to beremotely copied, a renewal process completion notice is issued to thehost computer of the main center immediately after the data renewalprocess for the disk subsystem in the main center is completed, tothereafter execute the data renewal (reflection) of the disk subsystemin the remote center, asynchronously with the data renewal in the maincenter. Since the data renewal is completed in a process time requiredby the main center, there is no transmission delay time or the like tobe caused by storing the data in the remote center.

In the asynchronous type, the data contents in a disk subsystem of theremote center are not always consistent with those in the main center.Therefore, if the function of the main center is lost by accidents orthe like, the data still not reflected upon the remote center is lost.However, an access performance of a disk subsystem in the main centercan be maintained at the level when the remote copy function is notexecuted.

In order to back up data so as not to be lost by natural disasters suchas earthquakes, it is necessary to set the distance between the main andremote centers to about 100 km to several tens km. Although it ispossible to use a high speed communication line, for example, of a 100Mbit/sec to 300 Mbit/sec class for the remote copy function, anexpensive line subscription fee is incurred upon a customer of the disksubsystem, and this approach is not economically suitable.

“Order Integrity”

There is another problem different from the above-described issue of thedata transmission time. Namely, if the remote center backs up the dataof a plurality of disk subsystems of the main center, there occurs anissue (order integrity) that disk subsystems are required to be inone-to-one correspondence. In asynchronous remote copy, it is inevitablethat reflection of renewal data in the remote center is delayed from thetime when an actual renewal process is executed in the main center.However, the order of renewal in the remote center is required to be thesame as that in the main center.

A database or the like is generally constituted of a main body of thedatabase and various log and control information directly associatedwith the main body. When data is renewed, not only the database mainbody but also the log and control information is renewed to maintain thesystem integrity. Therefore, if the renewal order is not kept, theintegrity of information regarding the renewal order is also lost, andat the worst the whole of the database may be destructed.

“Involvement of Host Computer”

In the asynchronous remote copy under general environments where themain and remote centers have a plurality of disk subsystems, when thehost computer instructs the disk subsystem to renew data, it is commonthat the host computer adds renewal order information such as a timestamp to the data to make the corresponding disk subsystem in the remotecenter execute a renewal data reflection process in accordance with theadded information.

According to the remote copy function disclosed, for example, in thepublication of JP-A-6-290125 (U.S. Pat. No. 5,446,871), generation andsupply of renewal order information and a renewal data reflectionprocess based upon this information are realized through the cooperationbetween the operating system of a host computer in a main center and itsdisk subsystems and the operating system of a host computer in a remotecenter and its disk subsystems.

SUMMARY OF THE INVENTION

This prior art can realize the asynchronous remote copy function whileensuring the renewal order between main and remote centers. With thisprior art, however, both the upper level software and a disk subsystemare required to have the mechanism for realizing the remote copyfunction, and also they are required to operate in cooperation. Sincenew custom software is required to be incorporated, a user is necessaryto perform works such as software incorporation, setting and check, andmodification of system designs to be caused by an increased CPU load.Incorporation of this conventional function is, therefore, associatedwith some obstacles such as a predetermined work period for suchpreparation and a cost therefor.

If the asynchronous remote copy function is executed when the capacityof the communication line to a remote center is not sufficient, renewaldata not reflected upon the remote center increases.

It is an object of the present invention to realize an asynchronous typeremote copy function capable of ensuring a renewal order and dataintegrity and facilitating its incorporation with less performancedegradation of a main center, by using only the function of a disksubsystem without incorporating new software.

It is another object of the present invention to realize a remote copyfunction without incurring an expensive line subscription fee upon acustomer of a disk subsystem, by applying an asynchronous type remotecopy function to the disk subsystem capable of storing a large amount ofdata.

Each of a main center and a remote center is provided with a disksubsystem serving as a gateway (hereinafter called a gateway subsystem)which is connected to a data transmission line. All disk subsystems inboth the centers to which a remote copy is executed, are connected tothe corresponding gateway subsystem of each center. A volume of the disksubsystem in the main center to be remotely copied and a desired volumeof the gateway subsystem in the main center are coupled by a synchronoustype remote function to mirror data. If a system process time delay orthe like can be neglected between the volume of the disk subsystem inthe main center to be remotely copied and the volume of the gatewaysubsystem in the main center, data consistency can be retained.

Data is mirrored between volumes of the gateway subsystems of the mainand remote centers through asynchronous remote copy. In this case, thegateway subsystem of the main center sends renewal data to the gatewaysubsystem of the remote center in accordance with the renewal order ofvolumes of disk subsystems of the main center, whereas the gatewaysubsystem of the remote center reflects the renewal data uponcorresponding volumes of the remote center in accordance with thereception order of the renewal data.

Data is mirrored between the volume of the gateway subsystem of theremote center and the volume of each disk subsystem through synchronousremote copy. Data same as from a macro viewpoint is always stored in thevolume of the gateway subsystem of the remote center and in the volumeof the disk subsystem to be remotely copied.

The gateway subsystem stores data in the volume to be remotely copied,in a buffer memory of the gateway subsystem. Since the gateway subsystemhas the buffer memory, generally an area for storing data in the gatewaysubsystem is not necessarily required. However, if there is an availablearea in the gateway subsystem, this area can be utilized for datatransfer via the transmission line, depending upon the capacity of thetransmission line.

With the above-described configuration, it is possible to mirror databetween a plurality of disk subsystems of the main center and aplurality of disk subsystems of the remote center, by using thefunctions of the disk subsystems, while the data renewal order isretained. Reflection of renewal data upon the remote center can beperformed asynchronously with the data renewal process at each disksubsystem of the main center. It is therefore possible to provide adisaster resistant back-up system of high performance and easyincorporation. Depending upon the communication capacity of thetransmission line, the storage area of the subsystem can be utilized sothat a burden of a line subscription fee of a customer can be reduced.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing the overall structure of a remote copysystem according to an embodiment of the invention.

FIG. 2 is a flow chart illustrating the details of an operation of theremote copy system.

FIG. 3 is a flow chart illustrating the details of an operation of theremote copy system to follow the flow chart of FIG. 2.

FIG. 4 is a flow chart illustrating the operation of a remote copysystem having a gateway subsystem provided with a buffer area.

FIG. 5 is a diagram showing the internal structure of a gatewaysubsystem.

DESCRIPTION OF THE EMBODIMENTS

An embodiment of the invention applied to a general computer system willbe described with reference to the accompanying drawings.

FIG. 1 shows an example of the structure of a system embodying thisinvention and allowing data to be mirrored between arbitrary two datacenters among a plurality of data centers each provided with a generalcomputer system.

One or a plurality of disk subsystems in a main center and one or aplurality of disk subsystems in a remote center are interconnected viagateway subsystems without involving the host computers, to realize aremote copy system for mirroring data between both the centers.

In the main center 12 shown in FIG. 1, a central processing unit (hostcomputer) 1 is connected via interface cables 2 to disk subsystems 3-1,3-2, . . . , 3-n. The disk subsystems 3-1, 3-2, . . . , 3-n store datato be referred to, or renewed, by the host computer 1. The gatewaysubsystem 5 is connected via interface cables 4 to the disk subsystems3-1 to 3-n.

The gateway subsystem 7 is provided in the remote center 13 andconnected via an interface cable 6 to the gateway subsystem 5 of themain center 12. The interface cable 6 is connectable to a generalcommunication line. In this embodiment, therefore, the interface cable 6is intended to include such a function.

When the host computer 1 issues a data write request to a disk subsystem3-1 or the like, the disk subsystem 3-1 or the like writes the data inits buffer memory. Synchronously with this timing the disk subsystem 3-1or the like issues a data write request to the gateway subsystem 5.

Upon reception of this write request, the gateway subsystem 5 writes thedata in its buffer memory. Asynchronously with the data write in thebuffer memory of the gateway subsystem 5, the gateway subsystem 5 issuesa data write request to the gateway subsystem 7 at the remote site. Itis essential to use a gateway subsystem 5 irrespective of how many disksubsystems 3-1 to 3-n are used.

The gateway subsystem 7 stores data supplied from the gateway subsystem5 in its buffer memory, in the order of data write requests. It isessential to use a gateway subsystem 7.

Disk subsystems 9-1, 9-2, . . . , 9-n are connected via interface cables8 to the gateway subsystem 7. When a data write request is issued fromthe main center 12 to the gateway subsystem 7, synchronously with thistiming the gateway subsystem 7 writes the data therein and in the disksubsystem 9-1.

Remote copy is therefore executed by sequentially issuing a writerequest to a subsystem and then to the next subsystem. When a data writerequest is issued from the host computer 1 to one or a plurality of disksubsystems 3-1 to 3-n, the same data is loaded in one or a plurality ofdisk subsystems 9-1 to 9-n of the remote center 13. Arrows shown in FIG.1 indicate the flow of the data instructed to be written by the hostcomputer 1.

In the remote center 13, the host computer 11 is connected via interfacecables 10 to the disk subsystems 9-1 to 9-n, and is a central processingunit which executes data reference and renewal relative to the disksubsystem 9-1 or the like. When the host computer 1 of the main center12 cannot provide its intrinsic functions because of disasters, systemfailures or the like, the host computer 11 can operate as an alternativeto the host computer 1. In addition, the host computer 11 can execute aprocess different from that of the host computer 1 of the main center 12by using data stored in the disk subsystem 9-1 or the like,independently from the operation of the host computer 1. However, if thehost computer 11 does not execute a process by using the disk subsystem9-1, this host computer 11 is unnecessary.

The outline of the data mirroring method and operation according to theembodiment of this invention will be described with reference to FIGS. 2and 3. A volume, a data set and a disk subsystem which store data to bemirrored are preselected by an administrator. The relation between thevolume, data set and disk subsystem in which the data is stored and avolume, a data set and a disk subsystem which store a copy of the data,is preset to the disk subsystems by the administrator from the hostcomputers.

In other words, a write destination of data to be copied is determinedsequentially between respective disk subsystems described above. Forexample, when data is written in a volume of the disk subsystem 3-1, itis set in such a manner that in which volume of the gateway subsystem 5the data is written from the disk subsystem 3-1, then in which volume ofthe gateway subsystem 7 the data is written from the gateway subsystem5, lastly in which volume of the disk subsystem 9-1 the data is writtenfrom the gateway subsystem 7. Such settings are provided for each disksubsystem including the gateway disk subsystem.

For such settings, the serial number and volume of each disk subsystemare used. For example, a volume A of the disk subsystem 3-1 is set to avolume B of the gateway subsystem 5, the volume B of the gatewaysubsystem 5 is set to a volume C of the gateway subsystem 7, and thevolume C of the gateway subsystem 7 is set to a volume D of the disksubsystem 9-1. In this manner, the data written in the volume A of thedisk subsystem 3-1 is copied to the volume D of the disk subsystem 9-1.This setting is conducted for all volumes.

Such preselect and preset may be effected by using a console or aservice processor without using the host computer, if the disk subsystemcan be connected to or provided with its console or service processor.The flow chart shown in FIG. 2 illustrates the operation assuming thatthe host computer is used for such preselect and preset. For suchpreset, a specific address indicating the volume or disk subsystem maybe designated, or the volume or disk subsystem in an arbitrary addressrange may be selected by using a control program in the disk subsystem.Path setting and pair setting are used as an example of initial setting(FIG. 2, 201).

The further description will be given with reference to the accompanyingdrawings.

As the host computer 1 (FIG. 1) issues a data write request (hereinaftercalled a write command) to the disk subsystem 3-1, 3-2, . . . , 3-n(211) (FIG. 2, 202), the disk subsystem 3-1, 3-2, . . . , 3-n executes adata load process for loading the write data therein in response to thewrite command, and also issues a write command for the data to thegateway subsystem 5 (212) (203). The write command is a command fortransferring an instruction of data write and the write data itself.

Upon reception of the write command, the gateway subsystem 5 executes aprocess corresponding to the write command (204). After the gatewaysubsystem completes a data load process for loading the data in itsbuffer memory, it notifies a process completion to the disk subsystem3-1, 3-2, . . . , 3-2 (211). A write command number is assigned to eachwrite command in the order of the process completion (205), and at thetiming determined basing upon the processing capability of the gatewaysubsystem 5, the write command assigned the write command number isissued to the gateway subsystem 7 (213) in the order of the writecommand number (206).

Under the conditions that the disk subsystem 3-1, 3-2, . . . , 3-ncompletes a process for the write command issued from the host computer1, i.e., completes a data load process for loading the data therein, andreceives a write process completion notice from the gateway subsystem 5(212) (221), the disk subsystem supplies a write command processcompletion notice to the host computer 1 (222).

The gateway subsystem 7 (213) confirms, from the write command numbersassigned to respective write commands issued from the gateway subsystem5 (212), whether the write commands have been received in the order ofthe assigned write command numbers. Thereafter, the gateway subsystemexecutes the processes corresponding to the write commands, i.e.,executes a data load process (301) for loading the data in its buffermemory. Thereafter, a write command corresponding to the loaded data isissued to the corresponding disk subsystem 9 (311) (302). Upon receptionof the write command issued from the gateway subsystem 7, the disksubsystem 9 (311) executes a process corresponding to the write command,i.e., executes a data load process for loading the data therein (303).

After the disk subsystem 9-1, 9-2, . . . , 9-n (311) completes theprocess corresponding to the write command, i.e., completes the dataload process for loading the data in its buffer memory, it supplies aprocess completion notice to the gateway subsystem 7 (321). Under theconditions that the gateway subsystem 7 (213) completes the data loadprocess for loading the data therein and receives the write processcompletion notice from the disk subsystem 9-1, 9-2, . . . , 9-n, thegateway subsystem 7 supplies a process completion notice for the writecommand to the gateway subsystem 5 (322).

According to the present invention, data written by the host computer 1is mirrored between the disk subsystem 3-1, 3-2, . . . , 3-n and thegateway subsystem 5 and is maintained consistent from a macro viewpoint.At this time, the gateway subsystem 5 adds information (serial number)to the data in order to hold the renewal order.

Data is mirrored between the gateway subsystems 5 and 7 throughasynchronous remote copy while the renewal order is ensured.Synchronously with the data renewal by the gateway subsystem 7, the disksubsystem 9-1, 9-2, . . . , 9-n renews the data. These operations areall realized by only the functions of the disk subsystem including adisk subsystem having the gateway function so that any load is notapplied to the processing performance of the host computer.

The operation of the remote computer system which uses a buffer area ofeach gateway subsystem when the communication capacity of thetransmission line is not sufficient, will be described with reference toFIG. 4. In FIG. 4, blocks having identical reference numerals to thoseshown in FIGS. 2 and 3 have been already described above. In thissystem, a buffer area for temporarily storing write data is provided ateach gateway subsystem in order to prevent an overflow of a buffermemory for a general transmission line. Data stored in the buffer areaof the subsystem is sent from the main center 12 to the remote center 13via the transmission line, and to the gateway subsystem via the bufferarea on the side of the remote center 12. Although time consistency formirroring is degraded, the asynchronous type remote copy function can berealized without using a high capacity communication line.

FIG. 5 shows the structure of the gateway subsystem 5. The structure ofthe gateway subsystem 7 is the same as the gateway subsystem 5.

The gateway subsystem 5 has: an interface control unit 11 for data(including information) transfer to and from the disk subsystem 3-1 orthe like and the gateway subsystem 7; a data buffer 12 for temporarilystoring the data; a magnetic disk drive 13 as a storage medium forstoring the data; a control memory for storing remote copy statusinformation (as to which volume of which disk subsystem is written towhich volume of the gateway subsystem 5, as to which volume of thegateway subsystem 5 is written in which volume of the gateway subsystem,and the like); a microprocessor 14 for controlling transfer of thesedata; a service processor panel 15 allowing a user to set how the remotecopy is executed; and a disk array subsystem control unit 17 forcontrolling these components. In this example, although the data buffer12 is provided at the gateway subsystem 12, this data buffer 12 is notnecessary if a cache memory capable of performing a similar function tothe data buffer 12 is provided, because the cache memory can function asthe data buffer 12. In this specification, therefore, the data buffer 12is intended to be inclusive of such a cache memory. Also in thisexample, although the control memory 16 is provided at the gatewaysubsystems, this control memory 16 is not necessary if a remote copycontrol information storage unit capable of performing a similarfunction to the control memory 16 is provided, because the remote copyinformation storage unit can function as the control memory 16. In thisspecification, therefore, the control memory 16 is intended to beinclusive of such a remote copy information storage unit.

As described so far, according to the present invention, an asynchronoustype remote copy system can be realized which can ensure the datarenewal order and data integrity by using the functions of disksubsystems without incorporating new software and which is easy to beincorporated and free from degradation of the process performance of themain center.

A storage area of the subsystem can be used depending upon thecommunication capacity of the transmission line so that a burden of aline subscription fee of a customer can be reduced.

1. A plurality of storage systems comprising: a first storage systemincluding a first controller controlling a first write request receivedfrom at least one host computer, and a first storage storing data undercontrol by said first controller; a second storage system including asecond controller controlling a second write request received from atleast one host computer, and a second storage storing data under controlby said second controller; a third storage system connected to saidfirst storage system and said second storage system, said third storagesystem comprises: a third controller controlling data received from saidfirst storage system synchronously and controlling data received fromsaid second storage system synchronously, and a third storage storingdata under control by said third controller; and a fourth storage systemconnected to said third storage system, said fourth storage systemcomprises: a fourth controller controlling data received from said thirdstorage system asynchronously, and a fourth storage storing data undercontrol by said fourth controller.
 2. A plurality of storage systemsaccording to claim 1, wherein said first controller of said firststorage system receives the first write request, sends a copy of datarequested by the first write request to said third storage system,stores data requested by the first write request into said firststorage, and receives an indication of completion of receipt of the copyof data requested by the first write request from said third controllerof said third storage system, and wherein said third controller of saidthird storage system receives the copy of data requested by the firstwrite request from said first storage controller of said first storagesystem, sends the indication of completion of receipt of the copy ofdata requested by the first write request to the first controller ofsaid first storage system, and stores the copy of data requested by thefirst write request into said third storage.
 3. A plurality of storagesystems according to claim 2, wherein said first controller of saidfirst storage system sends a copy of data requested by the first writerequest to said third storage system, after said first controller ofsaid first storage system stores data requested by the first writerequest into said first storage.
 4. A plurality of storage systemsaccording to claim 2, wherein said third controller of said thirdstorage system sends the indication of completion of receipt of the copyof data requested by the first write request to said first controller ofsaid first storage system, after said third controller of said thirdstorage system stores the copy of data requested by the first writerequest into said third storage.
 5. A plurality of storage systemsaccording to claim 2, wherein said first controller of said firststorage system sends an indication of completion of processing, aftersaid first controller of said first storage system receives theindication of completion of receipt of the copy of data requested by thefirst write request from said third controller of said third storage. 6.A plurality of storage systems according to claim 1, wherein said secondcontroller of said second storage system receives the second writerequest, sends a copy of data requested by the second write request tosaid third storage system, stores data requested by the second writerequest into said second storage, and receives an indication ofcompletion of receipt of the copy of data requested by the second writerequest from said third controller of said third storage system, andwherein said third controller of said third storage system receiving thecopy of data requested by the second write request from said secondstorage controller of said second storage system, and sends theindication of completion of receipt of the copy of data requested by thesecond write request to the second controller of said second storagesystem, and stores the copy of data requested by the second writerequest into said third storage.
 7. A plurality of storage systemsaccording to claim 6, wherein said second controller of said secondstorage system sends a copy of data requested by the second writerequest to said third storage system, after said second controller ofsaid second storage system stores data requested by the second writerequest into said second storage.
 8. A plurality of storage systemsaccording to claim 6, wherein said third controller of said thirdstorage system sends the indication of completion of receipt of the copyof data requested by the second write request to the second controllerof said second storage system, after said third controller of said thirdstorage system stores the copy of data requested by the second writerequest into said third storage.
 9. A plurality of storage systemsaccording to claim 6, wherein said second controller of said secondstorage system sends an indication of completion of processing, aftersaid second controller of said second storage system receives theindication of completion of receipt of the copy of data requested by thesecond write request from said third controller of said third storage.10. A plurality of storage systems according to claim 1, wherein saidthird controller of said third storage system receives data from saidfirst storage system, sends a copy of data received from the firststorage system to said forth storage system, stores data received fromthe first storage system into said third storage, and receives anindication of completion of receipt of the copy of data received fromthe first storage system, from said fourth controller of said fourthstorage system, and wherein said forth controller of said fourth storagesystem receives the copy of data received from the first storage system,from said third storage controller of said third storage system, sendsan indication of completion of receipt of the copy of data received fromthe first storage system to the third controller of said third storagesystem, and stores the copy of data received from the first storagesystem into said fourth storage.
 11. A plurality of storage systemsaccording to claim 10, wherein said third controller of said thirdstorage system sends a copy of data received from the first storagesystem to said fourth storage system, after said third controller ofsaid third storage system stores data received from the first storagesystem into said third storage.
 12. A plurality of storage systemsaccording to claim 10, wherein said forth controller of said fourthstorage system sends the indication of completion of receipt of the copyof data received from the first storage system to the third controllerof said third storage system, after said forth controller of said fourthstorage system stores the copy of data received from the first storagesystem into said fourth storage.
 13. A plurality of storage systemsaccording to claim 10, wherein said third controller of said thirdstorage system stores data received from the first storage system intosaid third storage, asynchronously with said first controller of saidfirst storage system sending an indication of completion of processing.14. A plurality of storage systems according to claim 10, wherein saidfourth controller of said fourth storage system stores the copy of datareceived from the first storage system into said forth storage,asynchronously with said first controller of said first storage systemsending an indication of completion of processing.
 15. A plurality ofstorage systems according to claim 10, wherein said third controller ofsaid third storage system receives data from said second storage system,sends a copy of data received from the second storage system to saidfourth storage system, stores data received from the second storagesystem into said third storage, and receives an indication of completionof receipt of the copy of data received from the second storage system,from said fourth controller of said fourth storage system, and whereinsaid fourth controller of said fourth storage system receives the copyof data received from the second storage system, from said third storagecontroller of said third storage system, and send an indication ofcompletion of receipt of the copy of data received from the secondstorage system to the third controller of said third storage system, andstores the copy of data received from the second storage system intosaid fourth storage.
 16. A plurality of storage systems according toclaim 15, wherein said third controller of said third storage systemsends a copy of data received from the second storage system to saidfourth storage system, after said third controller of said third storagesystem stores data received from the second storage system into saidthird storage.
 17. A plurality of storage systems according to claim 15,wherein said fourth controller of said forth storage system sends theindication of completion of receipt of the copy of data received fromthe second storage system to the third controller of said third storagesystem, after said fourth controller of said forth storage system storesthe copy of data received from the second storage system into saidfourth storage.
 18. A plurality of storage systems according to claim15, wherein said third controller of said third storage system storesdata received from the second storage system into said third storage,asynchronously with said second controller of said second storage systemsending of an indication of completion of processing.
 19. A plurality ofstorage systems according to claim 15, wherein said fourth controller ofsaid fourth storage system stores the copy of data received from thesecond storage system into said fourth storage, asynchronously with saidsecond controller of said second storage system sending of an indicationof completion of processing.